Problem: there's no compact historical state storage #722

yihuang · 2022-09-27T06:29:45Z

Closes: #704
Solution:

Integration version store and streaming service.

👮🏻👮🏻👮🏻 !!!! REFERENCE THE PROBLEM YOUR ARE SOLVING IN THE PR TITLE AND DESCRIBE YOUR SOLUTION HERE !!!! DO NOT FORGET !!!! 👮🏻👮🏻👮🏻

PR Checklist:

Thank you for your code, it's appreciated! :)

app/app.go

github-advanced-security

Found 14 potential problems in the proposed changes. Check the Files changed tab for more details.

github-advanced-security

Found 14 potential problems in the proposed changes. Check the Files changed tab for more details.

app/app.go

versiondb/streaming_service.go

github-advanced-security

Found 14 potential problems in the proposed changes. Check the Files changed tab for more details.

codecov · 2022-09-27T11:43:19Z

Codecov Report

Merging #722 (f46daef) into main (be8d53d) will increase coverage by 0.59%.
The diff coverage is 35.81%.

@@            Coverage Diff             @@
##             main     #722      +/-   ##
==========================================
+ Coverage   33.99%   34.59%   +0.59%     
==========================================
  Files          28       37       +9     
  Lines        1506     2246     +740     
==========================================
+ Hits          512      777     +265     
- Misses        941     1388     +447     
- Partials       53       81      +28

Impacted Files	Coverage Δ
versiondb/backend_test_utils.go	`0.00% <0.00%> (ø)`
versiondb/multistore.go	`0.00% <0.00%> (ø)`
versiondb/store.go	`0.00% <0.00%> (ø)`
versiondb/streaming_service.go	`0.00% <0.00%> (ø)`
versiondb/utils.go	`0.00% <0.00%> (ø)`
versiondb/tmdb/history.go	`58.13% <58.13%> (ø)`
versiondb/tmdb/store.go	`68.00% <68.00%> (ø)`
versiondb/tmdb/iterator.go	`88.00% <88.00%> (ø)`
versiondb/sync.go	`91.66% <91.66%> (ø)`

mmsqe · 2022-09-28T02:47:53Z

versiondb/tmdb/store.go

+	if err := historyBatch.WriteSync(); err != nil {
+		return err
+	}
+	return plainBatch.WriteSync()


will we have dirty data if one WriteSync fails?

It happens in end blocker, so it's before the chain state commit, if any error happens here, the block is not committed, when restart, the block will be replayed and the versiondb is written again for the same block.
The writes here should be idempotent, so it should be fine I think.

It'll be in an inconsistency state before the block is replayed though. Any ideas how to check the inconsistency? maybe write a block number into the plain db.

oh if eventually consistent, maybe just only allow query after block get replayed?

we can write a block number into plain db, that'd be the latest height, and we don't support query height higher than that.

https://github.com/crypto-org-chain/cronos/pull/722/files#diff-aa83981ed9bef3f7410fa0d5177ca4934c6e4b3c7280205511237da9b3dc02abR92
you've updated the latest version, so only if plainBatch.WriteSync() succeeded and it will be updated. Right?

yes, a batch write is atomic.

app/app.go

github-advanced-security

Found 13 potential problems in the proposed changes. Check the Files changed tab for more details.

yihuang · 2022-09-28T06:58:12Z

Some numbers get from local testing with rocksdb:

	Space Amplification (`file size / data size`)	Data Size¹	File Size²	KV Pairs³	(manual compaction) Space Amplification (file size)
Iavl	0.77	3368379952	2578108675	27994442	0.73 (2465786706)
VersionDB	0.84	566304124	473096919	5792002	0.79 (449149073)
- change set	0.82	536711835	438836315	5591300	0.8 (430133115)
- history index	0.54	20596897	11088590	100353	0.54 (11135326)
- plain db	2.58	8995392	23172014	100349	0.88 (7880632)

The ratio of versiondb/iavl file size reduction is: 0.18

Sum of sizes of all key value pairs ↩
Sum of sizes of db files. ↩
Number of key value pairs. ↩

yihuang · 2022-09-29T07:50:40Z

Application.db	File Size	KV Pairs	Data Size	Space Amplification
Rocksdb, archive	4.9G	31938977	3.7G	1.3
Rocksdb, prune=everything	438M	1481079	182.8M	2.4
MDBX, archive	6.5G			1.8
MDBX, prune=everything	735M			4

VersionDB	File Size	KV Pairs	Data Size	Space Amplification
Rocksdb
- Change set	153.5M	2082966	190.6M	0.8
- History index	39M	300456	30.9M	1.26
- Plain state	27.4M	300452	25.6M	1.07
MDBX
- Change set	218M			1.14
- History index	72M			2.4
- Plain state	62M			2.4

Just run some more tests to compare mdbx and rocksdb in terms of db size.
Here we don't use the fancy features of mdbx like dupsort, just use the normal get/set/cursor apis to fit into the tm-db interface.
So in terms of db size, rocksdb is not bad, it's kind of expected since mdbx don't do compression at all, and we don't use dupsort to compress the key prefixes. It's possible to optimize for mdbx in versiondb implementation though.

github-advanced-security

gosec found more than 10 potential problems in the proposed changes. Check the Files changed tab for more details.

JayT106 · 2022-10-07T19:50:02Z

versiondb/tmdb/store.go

+	if latest == height {
+		return s.plainDB.Get(rawKey)
+	}
+


maybe return nil or error here if height > latest ? what's the expected behavior in this case?

= Sounds good

I added a check, so it'll fail if height is in the future.

JayT106 · 2022-10-07T19:53:06Z

versiondb/tmdb/store.go

+		return s.plainDB.Get(rawKey)
+	}
+
+	found, err := SeekHistoryIndex(s.historyDB, rawKey, uint64(height))


any chance found will be equal to 0?

I think no, it only finds height larger than target.

versiondb/multistore.go

+	for k, v := range s.transientStores {
+		stores[k] = v
+	}


versiondb/multistore.go

+	for _, key := range keys {
+		s.transientStores[key] = transient.NewStore()
+	}


versiondb/multistore.go

+	for _, key := range keys {
+		s.transientStores[key] = mem.NewStore()
+	}


versiondb/multistore.go

Closes: crypto-org-chain#704 Solution: - Integration version store and streaming service. multiple backends db size benchmark support mdbx backend through tm-db store latest version support GetLatestVersion query multistore test versiondb streamer fix lint fix state listener temp fix state listening optimize common case fix lint try to fix mdbx build update cosmos-sdk fix clone append add test case fix subkey problem in history index revert chunking, hard to work with variable length key support iterator check future height fix lint new state listener fix latest state in query fix integration test fix prune node test update dependency add utility to read from file streamer Update versiondb/multistore.go Signed-off-by: yihuang <huang@crypto.com> add unit test create common backend test cases update dependency update with new file streamer format Problem: python3.10 is not used in integration tests Solution: - start using python3.10, prepare to later PRs which need the new features - update nixpkgs nesserary for the nix stuff to work. python-roaring64 remove debug log add test cases, improve coverage

app/app.go

+			for _, storeKey := range keys {
+				exposeStoreKeys = append(exposeStoreKeys, storeKey)
+			}


mmsqe · 2022-12-07T09:03:46Z

versiondb/tmdb/store.go

+	changesetBatch := s.changesetDB.NewBatch()
+	defer changesetBatch.Close()
+
+	for _, pair := range changeSet {


will we have big changeSet, which need allocate capacity for key for reduce?

integration_tests/test_benchmark_storage.py

mmsqe · 2022-12-08T07:34:29Z

integration_tests/configs/cosmovisor_gravity.jsonnet

@@ -5,6 +5,7 @@ config {
    'cmd-flags': '--unsafe-experimental',
    'app-config'+: {
      'minimum-gas-prices': '100000000000basetcro',
+      store:: super.store,


might need rm store for these two as well

only need this for upgrade test, because it's not supported in previous version.

mmsqe · 2022-12-09T02:38:47Z

versiondb/dbutils.go

+	"github.com/RoaringBitmap/roaring/roaring64"
+)
+
+var ChunkLimit = uint64(1950) // threshold beyond which MDBX overflow pages appear: 4096 / 2 - (keySize + 8)


do we also keep this chunk?

Co-authored-by: mmsqe <tqd0800210105@gmail.com> Signed-off-by: yihuang <huang@crypto.com>

yihuang · 2022-12-27T04:07:52Z

there's a better approach to use user-timestamp feature of rocksdb v7: #791

github-advanced-security bot found potential problems Sep 27, 2022

View reviewed changes

app/app.go Fixed Show fixed Hide fixed

yihuang force-pushed the version-store branch from cc3ac31 to 760f02a Compare September 27, 2022 06:34

github-advanced-security bot found potential problems Sep 27, 2022

View reviewed changes

yihuang force-pushed the version-store branch from 760f02a to f6816f2 Compare September 27, 2022 06:39

github-advanced-security bot found potential problems Sep 27, 2022

View reviewed changes

app/app.go Fixed Show fixed Hide fixed

versiondb/streaming_service.go Fixed Show fixed Hide fixed

github-advanced-security bot found potential problems Sep 27, 2022

View reviewed changes

mmsqe reviewed Sep 28, 2022

View reviewed changes

yihuang commented Sep 28, 2022

View reviewed changes

app/app.go Outdated Show resolved Hide resolved

github-advanced-security bot found potential problems Sep 28, 2022

View reviewed changes

github-advanced-security bot found potential problems Oct 6, 2022

View reviewed changes

yihuang force-pushed the version-store branch from 0d141b1 to ea207ed Compare October 6, 2022 02:23

yihuang mentioned this pull request Oct 6, 2022

Store plain historical states to reduce the db size of archive node cosmos/cosmos-sdk#13317

Closed

yihuang force-pushed the version-store branch from 0389aab to 65035a9 Compare October 7, 2022 07:07

JayT106 reviewed Oct 7, 2022

View reviewed changes

yihuang force-pushed the version-store branch from 12c900a to 085ebd5 Compare October 10, 2022 04:32

github-advanced-security bot found potential problems Oct 12, 2022

View reviewed changes

yihuang force-pushed the version-store branch from a246214 to 7c396f3 Compare October 13, 2022 01:44

yihuang commented Oct 13, 2022

View reviewed changes

versiondb/multistore.go Outdated Show resolved Hide resolved

yihuang mentioned this pull request Nov 17, 2022

fix: state listener observe writes at wrong time cosmos/cosmos-sdk#13516

Merged

19 tasks

yihuang added 2 commits November 28, 2022 00:00

fix streamer config

7d1201a

yihuang force-pushed the version-store branch from 936cc90 to 7d1201a Compare November 28, 2022 08:17

fix streamer decoding

66e50d4

yihuang mentioned this pull request Nov 28, 2022

set working directory for node processes in supervisor crypto-com/pystarport#78

Merged

fix linters

850ed66

yihuang force-pushed the version-store branch from 4c1f751 to 850ed66 Compare November 28, 2022 08:38

github-advanced-security bot found potential problems Nov 28, 2022

View reviewed changes

app/app.go

Comment on lines +384 to +386

for _, storeKey := range keys {

exposeStoreKeys = append(exposeStoreKeys, storeKey)

}

Check warning

Code scanning / CodeQL

Iteration over map

Iteration over map may be a possible source of non-determinism

yihuang added 2 commits November 28, 2022 18:00

home path

1f6eebb

fix integration test

856b18f

yihuang force-pushed the version-store branch from 3285914 to 856b18f Compare November 28, 2022 10:18

yihuang added 3 commits November 28, 2022 18:34

fix unit test

7b93785

fix upgrade test

234ad32

update to upstream

14be5b6

yihuang marked this pull request as ready for review December 7, 2022 08:45

yihuang requested a review from a team as a code owner December 7, 2022 08:45

yihuang requested review from mmsqe and leejw51crypto and removed request for a team December 7, 2022 08:45

yihuang added 3 commits December 7, 2022 16:45

Merge branch 'main' into version-store

3af8413

fix gravity test

b686422

Merge remote-tracking branch 'fork/version-store' into version-store

8e981c4

mmsqe approved these changes Dec 8, 2022

View reviewed changes

yihuang added 2 commits December 9, 2022 10:31

revert mdbx from this PR

eef8781

remove FIXME

611e2f3

mmsqe reviewed Dec 9, 2022

View reviewed changes

yihuang and others added 3 commits December 9, 2022 10:41

remove unused functions

bde66c1

Update integration_tests/test_benchmark_storage.py

659a562

Co-authored-by: mmsqe <tqd0800210105@gmail.com> Signed-off-by: yihuang <huang@crypto.com>

remove TODO

f46daef

yihuang marked this pull request as draft December 14, 2022 03:05

yihuang closed this Dec 27, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Problem: there's no compact historical state storage #722

Problem: there's no compact historical state storage #722

yihuang commented Sep 27, 2022

github-advanced-security bot left a comment

github-advanced-security bot left a comment

github-advanced-security bot left a comment

codecov bot commented Sep 27, 2022 •

edited

Loading

mmsqe Sep 28, 2022

yihuang Sep 28, 2022 •

edited

Loading

mmsqe Sep 29, 2022

yihuang Oct 6, 2022

JayT106 Oct 7, 2022

yihuang Oct 12, 2022 •

edited

Loading

github-advanced-security bot left a comment

yihuang commented Sep 28, 2022 •

edited

Loading

yihuang commented Sep 29, 2022 •

edited

Loading

github-advanced-security bot left a comment

JayT106 Oct 7, 2022 •

edited

Loading

yihuang Oct 8, 2022

yihuang Oct 12, 2022

JayT106 Oct 7, 2022

yihuang Oct 8, 2022

mmsqe Dec 7, 2022

mmsqe Dec 8, 2022

yihuang Dec 9, 2022

mmsqe Dec 9, 2022

yihuang Dec 9, 2022

yihuang commented Dec 27, 2022 •

edited

Loading

Problem: there's no compact historical state storage #722

Problem: there's no compact historical state storage #722

Conversation

yihuang commented Sep 27, 2022

PR Checklist:

github-advanced-security bot left a comment

Choose a reason for hiding this comment

github-advanced-security bot left a comment

Choose a reason for hiding this comment

github-advanced-security bot left a comment

Choose a reason for hiding this comment

codecov bot commented Sep 27, 2022 • edited Loading

Codecov Report

Choose a reason for hiding this comment

yihuang Sep 28, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yihuang Oct 12, 2022 • edited Loading

Choose a reason for hiding this comment

github-advanced-security bot left a comment

Choose a reason for hiding this comment

yihuang commented Sep 28, 2022 • edited Loading

Footnotes

yihuang commented Sep 29, 2022 • edited Loading

github-advanced-security bot left a comment

Choose a reason for hiding this comment

JayT106 Oct 7, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yihuang commented Dec 27, 2022 • edited Loading

codecov bot commented Sep 27, 2022 •

edited

Loading

yihuang Sep 28, 2022 •

edited

Loading

yihuang Oct 12, 2022 •

edited

Loading

yihuang commented Sep 28, 2022 •

edited

Loading

yihuang commented Sep 29, 2022 •

edited

Loading

JayT106 Oct 7, 2022 •

edited

Loading

yihuang commented Dec 27, 2022 •

edited

Loading